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In  contemporary  systems  the  functional  state  of  the  operator  is  not  considered  during  system 
operation.  Degraded  states  of  operator  functioning  can  result  from  the  demands  of 
controlling  complex  systems,  the  work  environment  and  internal  operator  variables.  This,  in 
turn,  can  lead  to  errors  and  overall  suboptimal  system  performance.  In  the  case  of  mental 
workload,  system  performance  could  be  improved  by  reducing  task  demands  during  periods  of 
operator  overload.  Accurate  estimation  of  the  operator’s  functional  state  is  crucial  to 
successful  implementation  of  an  adaptive  aiding  system.  One  method  of  determining 
operator  functional  state  is  by  monitoring  the  operator’s  physiology.  In  the  present  study, 
physiological  signals  were  used  to  continuously  monitor  subject’s  functional  state  and  to 
adapt  the  task  by  reducing  the  number  of  subtasks  when  high  levels  of  mental  workload  were 
detected.  The  goal  was  to  demonstrate  performance  improvement  with  adaptive  aiding. 
Because  adaptive  aiding  during  high  mental  workload  has  not  been  previously  implemented  its 
benefit  has  not  be  demonstrated.  Application  of  adaptive  aiding  techniques  reduced  tracking 
task  error  by  44%  and  resource  monitoring  error  by  33%.  These  results  demonstrate  the 
utility  of  adaptive  aiding  using  physiological  measures  with  artificial  neural  networks  to 
determine  the  appropriate  time  to  introduce  the  aiding. 


INTRODUCTION 

Today’s  complex  systems  can  place  high 
cognitive  demands  upon  their  operators.  The  rate  of 
information  flow,  complex  nature  of  this  information, 
and  the  required  decisions  can  become  overwhelming 
to  the  human  operator.  On  the  other  hand, 
automation  of  tasks  can  lead  to  complacency  and 
errors  of  inattention.  Current  systems  are  capable  of 
modifying  themselves  to  meet  the  current  needs  of 
the  operator.  This  includes  taking  over  some 
functions  for  the  operator  until  the  mental  load  is 
reduced.  In  other  cases  systems  can  adjust  in  order  to 
engage  the  operator  to  relieve  boredom  or 
inattention.  In  each  case  the  critical  need  is  the 
accurate  and  reliable  assessment  of  operator  state. 
Adaptive  aiding  by  the  system  can  be  beneficial  only 
when  supplied  at  the  appropriate  time  and  with  the 
consent  of  the  operator. 

One  method  of  determining  operator  functional 
state  is  by  monitoring  the  operator’s  physiology. 
Numerous  examples  of  physiological  measures 
providing  operator  state  assessment  are  available.  It 
has  been  demonstrated  that  the  various  physiological 
measures  provide  unique  information  about  several 
aspects  of  operator  state.  Eye  blink  rate  contains 
valuable  information  about  the  visual  demands  of 
tasks.  Heart  rate  is  of  use  to  determine  the  overall 
engagement  of  an  operator  by  a  task  (Wilson  & 
Eggemeier,  1991).  Because  of  the  nature  of 


physiological  signals  they  are  always  present  and 
thereby  provide  continuous  information  about 
operator  state. 

Several  studies  have  used  physiological  measures  to 
classify  operator  state  related  to  mental  workload. 
Most  of  these  studies  employed  electroencephalo- 
graphic  (EEG)  data  These  studies  used  either  simple, 
single  task  paradigms  (Gevins,  Smith,  Leong,  McEvoy, 
Whitfield,  Du  and  Rush,  1998;  Gevins  &  Smith,  1999; 
Nikolaev,  Ivanitskii  and  Ivanitskii,  1998;  Russell  & 
Wilson,  1998;  Wilson  &  Fisher,  1995)  or  used 
relatively  few  peripheral  physiological  variables  with 
complex  task  performance  (Wilson  &  Fisher,  1991). 
They  report  overall  successful  task  classification  in 
the  high  80  to  90  percent  correct  range.  These 
excellent  results  classifying  high  mental  workload  or 
altered  operator  state  are  very  encouraging.  They 
suggest  that  these  methods  could  be  used  to  provide 
accurate  and  reliable  operator  state  assessment  to  an 
adaptive  aiding  system.  Psychophysiological  measures 
have  been  used  to  implement  adaptive  aiding  in 
laboratory  situations  of  lowered  operator  engagement 
(Freeman,  Mikulka,  Prinzel.  &  Scerbo,  1999;  Pope, 
Comstock,  Bartolome,  Bogart  and  Burdette,  1995; 
Prinzel,  Scerbo,  Freeman  and  Mikula,  1997).  These 
investigations  showed  enhanced  operator  performance 
when  the  adaptive  aiding  system  detected  subject 
disengagement  and  modified  the  task  to  increase 
subject  involvement. 
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In  the  case  of  mental  overload,  system 
performance  could  be  improved  by  reducing  task 
demands.  Accurate  estimation  of  the  operator’s 
functional  state  is  crucial  to  successful  implementation 
of  an  adaptive  aiding  system.  In  the  present 
investigation  physiological  signals  were  continuously 
monitored  in  order  to  determine  the  subject’s 
functional  state  and  to  adapt  the  task  when  high  levels 
of  mental  workload  were  detected.  Adaptive  aiding 
during  periods  of  high  mental  workload  has  not  been 
accomplished  before  using  physiological  measures. 

The  goal  of  the  present  study  was  to  determine  the 
effects  of  adaptive  aiding  upon  task  performance 
when  the  aiding  was  provided  during  periods  of  high 
mental  workload.  While  improved  scores  are 
expected  it  is  possible  that  the  aiding  may  interfere 
with  subject  performance. 

METHODS 

Seven  subjects  (4  female)  participated  in  the 
experiment.  Their  age  range  was  from  19  to  26  years. 
They  were  trained  to  stable  performance  on  the 
NASA  Multiple  Attribute  Task  Battery  (MATB, 
Comstock  &  Arnegard,  1992).  After  initial 
familiarization  with  the  task  they  were  trained  to  use 
their  right  hand  to  control  a  j  oystick  and  their  left 
hand  to  control  a  mouse.  All  of  the  MATB  subtasks 
were  used,  lights  and  dials  monitoring,  tracking, 
resource  management  and  the  auditory  task.  Two 
levels  of  difficulty  were  used  and  was  manipulated  by 
varying  the  number  of  events  that  occurred  during 
each  of  the  five  minute  trials.  Performance  scores 
from  each  task  were  recorded  and  practice  was 
maintained  until  each  subject  exhibited  stable 
performance  on  all  tasks.  This  required 
approximately  six  hours  of  practice  spread  over  three 
days. 

Physiological  data  were  recorded  during  task 
performance.  The  physiological  data  consisted  of  six 
EEG  channels,  electrocardiographic  (ECG), 
electrooculographic  (EOG)  and  respiration  inputs.  On 
the  day  of  data  collection  the  subjects  practiced  the 
tasks  for  5  minutes.  Then  three  conditions  were 
presented  to  the  subjects.  The  first  was  a  resting 
condition  during  which  the  subjects  merely  looked  at 
the  screen  .  The  second  condition  required  them  to 
perform  the  task  at  the  low  workload  level.  During 
the  third  condition  the  task  was  presented  at  the 
higher  difficulty  level.  Using  the  physiological  data 
from  these  three  conditions  an  artificial  neural 
network  (ANN)  was  trained  to  recognize  these  three 
conditions.  The  input  to  the  ANN  consisted  of 
spectral  EEG  features  including  delta  (1-3  Hz),  theta 
(4-7  Hz),  alpha  (8  -  13  Hz)  and  beta  (14  -  30  Hz) 
bands.  Other  features  included  ECG  interbeat  intervals, 
blink  rate  and  blink  closure  duration,  and  respiration 


rate.  One  half  of  the  data  from  each  of  the  three 
conditions  were  randomly  selected  and  used  as  training 
data.  The  remaining  half  was  used  as  test  data  to 
determine  the  accuracy  of  the  ANN. 

The  trained  ANN  was  used  to  determine,  on-line, 
which  of  the  three  conditions  the  subjects  was 
performing  based  entirely  on  the  physiological  data. 
The  three  conditions  used  for  training  were  repeated 
twice.  During  both  replications,  on-line  determination 
of  subject  workload  level  was  performed  each  second. 
In  the  first  case  the  workload  classifications  were 
recorded  to  determine  the  accuracy  of  the  trained 
ANN.  In  the  second  case  adaptive  automation  was 
applied.  When  the  high  workload  condition  was 
detected  by  the  trained  ANN  the  MATB  task  was 
adapted  by  “turning  off’  two  of  the  subtasks.  During 
adaptive  aiding,  the  lights  and  dials  monitoring  and  the 
auditory  task  were  “turned  off’  and  their  areas  on  the 
screen  were  highlighted  in  blue  to  indicate  that  an 
aiding  period  was  in  progress.  The  subjects  were 
instructed  to  ignore  these  tasks  and  concentrate  their 
efforts  on  the  tracking  and  resource  management 
tasks.  They  were  given  practice  ‘ignoring’  these 
tasks. 

MATB  performance  scores  were  recorded  so  that 
the  effects  of  the  adaptive  aiding  could  be  evaluated. 

RESULTS 

The  accuracy  of  the  trained  ANN  was  first  tested 
by  having  it  classify  the  test  set  of  the  original  data 
run.  The  ANN  accuracy  for  the  training  data  was 
98.5%  correct.  This  almost  perfect  accuracy  of 
classifying  the  baseline,  low  and  high  workload 
conditions  is  typical.  During  the  first  test  run  where 
the  ANN  was  used  to  only  classified  the  workload, 
very  high  levels  of  classification  accuracy  were  also 
achieved.  The  accuracy’s  were  84.9%  for  the 
baseline,  82.0%  for  the  low  workload  and  86%  for  the 
high  workload  conditions.  See  Table  1.  These  results 
demonstrate  that  an  ANN  can  produce  high  levels  of 
correct  classification  while  subjects  perform  complex 
multiple  task  scenarios.  Noteworthy  from  the  table  is 
the  pattern  of  confusion  by  the  ANN  of  adjacent 
workload  conditions  when  misclassifications  did  occur. 
During  the  baseline  condition  the  vast  majority  of 
errors  were  assigned  to  the  low  condition.  The  errors 
during  the  low  condition  were  primarily  confusion 
with  the  baseline  condition  and  most  of  the  errors  for 
the  high  condition  were  miss  categorized  as  belonging 
to  the  low  condition. 


Classification  accuracies  for  each  subject  showed  an 
overall  mean  correct  classification  range  from  69.0% 
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to  97.8%.  See  Table  2.  The  highest  accuracy  for  any 
one  condition  was  100%.  All  of  the  observed 
accuracies  were  well  above  the  expected  chance  level 
of  33%. 


During  adaptive  aiding  the  subjects  performance 
on  the  tracking  and  resource  management  tasks  was 
monitored.  Because  the  purpose  of  the  aiding  is  to 
permit  the  subjects  to  concentrate  on  these  two  tasks 
during  epochs  of  high  workload  one  would  expect 
improved  performance.  By  removing  the  monitoring 
and  audio  tasks  the  subjects  were  free  to  focus  their 
efforts  on  the  remaining  two  tasks.  This  was  exactly 
the  case.  Adaptive  aiding  resulted  in  a  44%  reduction 
in  RMS  tracking  error  (t  =  -6.134,  p  <  0.0008) 
compared  to  the  nonadaptive  condition.  Performance 
on  the  resource  management  task  improved  with  a 
33%  reduction  in  the  error  score  that  was  marginally 
significant  (t  =  -1.822,  p  <  0.06). 

DISCUSSION 

These  results  demonstrate  that  an  ANN  using 
central  and  peripheral  nervous  system  features  can  be 
trained  to  very  accurately  determine,  on-line,  the 
functional  state  of  an  operator.  This  is  especially 
significant  in  light  of  the  complex,  multiple  task  that 
was  performed  by  the  subjects.  The  mean  correct 
classification  across  subjects  ranged  from  82.0%  to 
86.0%.  These  results  are  consistent  with  previous 
reports  and  demonstrate  the  high  levels  of  accuracy 
that  are  possible  using  ANNs.  No  doubt  using  both 
central  and  peripheral  nervous  system  features 
enhanced  the  performance  of  the  ANN.  Additional 
accuracy  may  be  achieved  by  including  performance 
features  when  possible  (Wilson  &  Russell,  1999). 

The  utilization  of  operator  state  information  to 
govern  the  application  of  adaptive  aiding  also 
provided  remarkable  results.  Lowering  task  demands 
based  upon 

operator  state  resulted  in  large  enhancements  in 
operator 

performance.  The  tracking  task  error  was  reduced  by 
44%  and  the  resource  management  error  was  reduced 
by 

33%.  Reduction  of  overall  task  demands  by 
temporarily 

removing  the  burden  of  the  monitoring  and  auditory 
task 


based  upon  the  physiologically  determined  operator 
state  freed  the  subjects  to  concentrate  on  the  two 
remaining  tasks  and  greatly  improve  their 
performance. 


The  effect  of  removing  the  two  subtasks  without 
regard  to  the  subject’s  state  remains  to  be  determined. 
Randomly  removing  these  subtasks  could  also  have 
improved  performance.  However,  it  is  possible  that 
there  would  have  been  no  change  or  degraded 
performance  from  the  random  removal  of  the  two 
subtasks.  Random  removal  might  interfere  with  the 
subject’s  strategy  and  lead  to  deteriorated 
performance.  This  question  will  have  to  be 
determined  by  further  research. 

Because  the  physiological  features  are 
continuously  available  on-line,  real  time  functional 
state  monitoring  systems  can  be  developed.  Further 
research  will  be  needed  to  find  the  best  methods  for 
determining  the  parameters  of  the  ANN  and  to 
enhanced  feature  selection.  Other  issues  still 
remaining  to  be  resolved  before  ANNs  can  be 
routinely  applied  in  the  work  place.  These  issues 
have  to  do  with  their  day  to  day  reliability  and  the 
necessity  to  establish  a  ANN  for  each  subject  rather 
that  using  a  generic  solution  that  would  accommodate 
all  operators. 

REFERENCES 

Comstock,  J.R.  and  Amegard,  R.J.  (1992).  The  multi¬ 
attribute  task  battery  for  human  operator  workload  and 
strategic  behavior  research.  NASA  Technical  Memorandum 
No.  104174. 

Eischeid,  T.M.,  Scerbo,  M.W.  and  Freeman  F.G.  (1998). 

The  effects  of  task  partitioning  and  computer  skill  on 
engagement  and  performance  with  an  adaptive,  Biocybemetic 
system..  Proceedings  of  the  Human  Factors  and  Ergonomics 
Society  42nd  Annual  Meeting.  133-137. 

Freedman,  F.G.,  Mikulka,  P.J.,  Prinzel,  L.J.  and  Scerbo, 
M.W.  (1999)  Evaluation  of  an  adaptive  automation  system 
using  three  EEG  indices  with  a  visual  tracking  task. 
Biological  Psychology.  50,  61-76. 

Gevins,  A.,  Smith,  M.E.,  Leong,  H.,  McEvoy,  L,  Whitfield, 
S.,  Du,  R.  and  Rush,  G.  (1998).  Monitoring  working 
memory  load  during  computer-based  tasks  with  EEG  pattern 
recognition  methods.  Human  Factors.  40,  79-91. 

Gevins,  A.  and  Smith,  M.E.  (1999).  Detecting  transient 
cognitive  impairment  with  EEG  pattern  recognition  methods. 
Aviation,  Space,  and  Environmental  Medicine.  70,  1018- 
1024. 


ASC00-0350  on  18  Feb  2000 


Nikolaev,  A.  R.,  Ivanitskii,  G.  A.  and  Ivanitskii,  A.M. 

(1998).  Reproducible  EEG  alpha-patterns  in  psychological 
task  solving.  Human  Physiology.  24,  261-268. 

Pope,  A.T.,  Bogart,  E.H.  and  Bartolome,  D.S.  (1995). 
Biocybemetic  system  evaluates  indices  of  operator  engagement 
in  automated  task.  Biological  Psychology.  40,  187-195. 

Russell,  C.A.  and  Wilson,  G.F.  (1998).  Air  traffic  controller 
functional  state  classification  using  neural  networks. 
Proceedings  of  the  Artificial  Neural  Networks  in  Engineering 

(ANNIE’981  Conference.  Vol  8,  649-654. 

Wilson,  G.  F.  &  Eggemeier,  F.  T.  (1991).  Physiological 
measures  of  workload  in  multi-task  environments  (pp.  329- 
360).  In  Damos,  D.  (Ed.)  Multiple-task  performance. 

London:  Taylor  and  Francis. 

Wilson,  G.  F.,  &  Fisher,  F.  (1991).  The  use  of  cardiac  and 
eye  blink  measures  to  determine  flight  segment  in  F4  crews. 
Aviation.  Space  and  Environmental  Medicine.  62,  959-961. 

Wilson,  G.  F.  &  Fisher,  F.  (1995)  Cognitive  task 
classification  based  upon  topographic  EEG  data.  Biological 
Psychology.  40,  239-250. 

Wilson,  G.F.  and  Russell,  C.  (1999).  Operator  Functional 
State  Classification  using  neural  networks  with  combined 
physiological  and  performance  features.  Proceedings  of  the 
Human  Factors  and  Ergonomics  Society  43rd  Annual 

Meeting.  1099-1102. 


